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Abstract 


In this contribution, 3D surveying, Artificial Intelligence and archival documentation are combined 
together for the documentation of different temporal states and deterioration processes of the in- 
terior walls of the Chapel of Sant'Agata in Pisa. The Chapel, located in the immediate vicinity of the 
Church of San Paolo a Ripa d'Arno, is of medieval origin. Its interior spaces have undergone a series 
of alterations, that modified over time the original layout and decorative apparatus. In the 1960s, an 
intense restoration activity brought to light the medieval frescoes of the Chapel, hidden under a thick 
layer of plaster. However, exposure to atmospheric agents and lack of maintenance have ruined those 
discovered walls, and only few traces of the ancient paintings are still visible today. 

Starting from these considerations and exploiting the elaboration of both survey data and existing 
documentation, the aim of this work is to compare the different states of conservation of the Chapel's 
frescoes in time, even relying on semantic classification techniques leveraging Artificial Intelligence. 
The result is a project of spatialization and classification of 2D (images) and/or 3D (mesh, point 
clouds) colorimetric data that can be applied to other case studies, to analyze the evolution of the 
painted surfaces related to other objects making part of the historical-architectural heritage. 
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Fig. 01.Texture-based 
approaches can be 
applied to images (a), 
ortho-photos (b) or UV 
maps (c). 


Introduction 


In recent years, the cultural heritage sector, on the one hand, has increasingly benefited from 
the use of incremental photogrammetric techniques for the spatialization of archival imagery 
[Bevilacqua et al. 2017; Manuel et al. 2013]. On the other hand, the exploitation of Artificial 
ntelligence (Al) techniques has allowed to improve and automate the interpretation of 
digital models [Fiorucci et al. 2020]. In application to 2D (images, orthophotos, UV maps) 
and 3D (point clouds, meshes) data, Machine Learning (ML) and its subset Deep Learning 
(DL) are emerging as effective tools to attribute semantic information to a set of otherwise 
unstructured data. The application of supervised ML approaches to the processing of survey 
data has indeed boosted some common semantic mapping processes, e.g., concerning the 
insertion of annotations related to degradation, pictorial elements or material types, by relying 
on a certain amount of training data while always pertaining a certain degree of supervision 
by an experienced operator. The so-called semantic segmentation process has so been 
understood as the reasoned subdivision of survey data on the basis of the more automated 
recognition of architectural components, degradation patterns or textures sharing common 
or at least similar features. Referring to a semi-automated semantic segmentation procedure 
for the enrichment of digital heritage data, purpose of this work is to analyze the evolution 
of the decorative apparatus of the remarkable case study of the Chapel of St. Agata in Pisa 
(figg. OI, 02) [I]. The result is a methodological approach that can be extended to other 
case studies, for the documentation of architectural objects characterized by a complex 
decorative and material apparatus, as well as by different temporal states that have altered 
over time the level of preservation, or even the presence, of some relevant elements and 
surfaces. 


State-of-the-art 


Recent developments of photogrammetry have revealed how digital 3D reconstructions 
can include archival documentation for the recovery of past architectural heritage. Metric 
knowledge can be extracted from archival data so to derive valuable information on the 
virtual restoration, anastylosis and reconstruction of destroyed or transformed objects, and 
also to compare previous and actual states of a building [Al Khalil, Grussenmeyer 2019; Croce 
et al. 2019]. Previous studies [Martinez Espejo Zaragoza et al. 2021; Bevilacqua et al. 2017] 
underlined the potentials of integrating existing surveying models with data coming from 
historic and archival photography, by appropriate spatialization of historic documentation 
within currently existing photogrammetric models. The calibration of the images and 
the orientation steps are fundamental in such pipelines, as the results are affected by the 
calculation of camera calibration information and by the appropriate detection of control 
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Fig. 02. Current in-plan 
configuration of the Cha- 
pel and spatial correlation 
with the Church of San 
Paolo a Ripa d'Arno. 
Adapted from the original 
drawing by L. Tenchini. 
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points between archival imagery-existing data [Condorelli, Rinaudo 2018]. If on the one 
hand historic images can be aligned on an already existing image sets (incremental alignment) 
[Pamart et al. 2019], on the other hand semantic segmentation techniques making the most 
of Al are being more and more exploited to improve photogrammetric models: Malinverni 
et al. [2019] tested several supervised and unsupervised strategies for the classification of 
architectural and archaeological scenes, and proved that supervised ML algorithms, as the 
RF generally outperform DL methods; Stathopoulou & Remondino [2019] considered the 
insertion of semantically labelled images to boost the dense reconstruction of a scene, in 
the pair selection, depth estimation and filtering phases. Murtiyoso & Grussenmeyer [2019] 
exploited deep learning-based semantic image segmentation and masking to create classified 
dense point clouds by exploitation of the photogrammetric process. A first effort towards the 
combination of historical photogrammetry with Al was presented by Condorelli & Rinaudo 
[2018], who experienced the use of DL methods for the metric reconstruction of no more 
existing buildings starting from historical film footages. However, the use of classified archival 
images to improve already created photogrammetric datasets has not been investigated yet 
but may be essential in supporting the visual and metric comparison between actual and 
previous states of a certain heritage object. 


The Chapel of St.Agata in Pisa 


The Chapel of St. Agata in Pisa, built on an octagonal plan, is located behind the apse of the 
Church of San Paolo a Ripa d'Arno, in Pisa. Its construction was ordered by the Vallombrosian 
monks of the adjacent church in the second half of the | Ith century; initially connected to 
the church by medieval buildings, the chapel is now surrounded by public greenery and 
closed by perimetral masonry walls articulated by pillars, three-mullioned windows and 
under-roof arches. It concludes upwards with a spire. In the 1960s, the restoration works of 
the interiors, entailing the removal of the outermost layer of plaster that covered the original 
walls, led to the discovery of frescoes dating back to the end of the | 3th century — beginning 
of the 14th century, and attributed to an anonymous Pisan master. Between the restoration 
works and today, a progressive loss of the discovered frescoes has occurred, even caused 
by the exposure of the chapel to the atmospheric agents. For this reason, although not 
being characterized by very large extension (in plan, the edges of the chapel are about 3 
meters long while the internal diameter is about 7 meters), the case study is relevant as 
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Fig. 03. Workflow of the 
proposed approach. 
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its investigation implies the comparison between the actual and previous condition of the 
building's internal space, in order to ensure the appropriate recognition of the parts of the 
brickwork that can be traced back to the original decorations. The Chapel is thus significative 
for the analysis of the integration between classification methods and archival imagery. 


Workflow: from 3D surveying to data elaboration and spatialization 


The data acquisition campaign for the Chapel of St. Agata was conducted back in the years 
2016 and 2017, in the framework of a joint research project [2] aiming at the metric and 
graphical documentation of the monument. The architectural surveying, involving both laser 
scanning and photogrammetric techniques, was performed to collect aesthetic and formal 
connotations of the object. 

As expressed by Bevilacqua et al. [2017] and Tenchini [2016] — to which the reader can refer 
for further details on the surveying process —, the results of the acquisition campaign were 
provided in the form of graphical documents at scale 1:25, and a first exploration on the 
integration of the surveying outputs with data derived from historic archival imagery was 
presented, towards the comprehension of actual/past conditions of the inner wall paintings 
of the Chapel. 

Starting from these survey data, and more specifically leveragingthe photogrammetric survey, 
a semantic classification is performed for the Chapel under study, for a more automated 
description of both: 


i) the current level of advanced degradation; 

ii) the alterations, in terms of material mapping and decay, that the interior walls of the 
Chapel have underwent in the period between the | 960s (restoration works) and today 
(current, existing state). 


The different phases of the workflow are summarized in fig. 03 and the related results 
are illustrated in the next sections. At first, the semantic classification is performed on 
UV maps or orthophotos obtained from the photographic campaign of years 2016-2017. 
The classes of degradation or material in which the photogrammetric model of the actual state 
is to be subdivided are identified on the digital support, by annotating a small portion of the 
ortho-photo. Then, a suitable set of features is chosen as additional stack of images that help the 
classification. Based on the provided data samples and on the extracted features, a supervised ML 
algorithm, the RF is trained to classify the remaining part of the dataset. 
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Fig. 04. Section of the 
chapel (a, drawing by L. 
Tenchini) and manual- 
ly-annotated training set 
with related legend (b). 


Fig. 05. Image adjustmen- 
ts, edge-detection and 
noise-reduction filters 
extracted from the input 
image. 
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To assess the classification result obtained, a confusion matrix and common performance metrics 
used in the Al domain are extracted from a validation set (accuracy, F-measure, precision and 
recall) [Matrone et al. 2020]. In a second phase, archival images (related to the wall paintings 
of the interior of the church found in the ‘60s) are in turn classified, considering once again 
a series of appropri- ate features and a suitable training set in order to distinguish, over the 
interior surface of the Chapel, the areas that previously exhibited traces of mural painting. 
The semantic segmentation is run in this case on the archival image, and the related classi- 
fication results (i.e., the classification information for the previous state) are finally projected 
back onto the photogrammetric model, by exploiting 2D/3D projective relationships. 

A comparison of the frescoes at the time they were found in the | 960s (archival and historic 
imagery for the previous state) and in the actual condition (photogrammetric model and 
images of the current state), respectively, can be derived in consequence. 


Traces of decorative paintings of the internal walls 


At first, the semi-automatic annotation method exploiting Al is performed solely considering the 
photogrammetric model of the current state of the Chapel. As for the example of fig. 04, the 
annotation of decorated and non-decorated parts is performed over a section of the building, 
derived from the elaboration of the ortho-mosaic of the photogrammetric model (fig. 04a). 

In the training phase, four annotation classes are manually identified on a portion of the ortho- 
photo and associated to dedicated colours and labels (fig. 04b): brick masonry still presenting 
traces of wall painting (class 0), chromatic alteration of brick masonry (1), plaster (2) and 
marble (3). The white background of the ortho-photo is excluded from the classification. 


Brightness 


Hessian trace Hessian eigenvalue 
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Fig. 06. Texture filters 
extracted from the input 
image. 


Fig. 07. Semantic segmen- 
tation results for a section 
of the Cappella di Sant'A- 
gata in Pisa, in opacity (a) 
and in transparency (b). 
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The portion of annotated data of fig. 04b constitutes a set of data samples used for learning. 
It is associated with the extraction of pertinent features, including image adjustments, edge- 
detection, and noise-reduction filters (fig. 05) as well as texture filters (fig. 06). 

The results of the automatic annotation by RF training are provided in fig. 07 and may be 
displayed in transparency over the input data. The distinction between parts of brick altered 
by the presence of plaster (chromatic alteration) and parts of brick where traces of the 
original frescoes are still recognizable is graphically highlighted by colour maps. In order 
to validate the classification results, the automatic annotation is compared to the manual 
annotation on a portion of the data (validation set of fig. 08a-08c). The graphical comparison 
between true and predicted labels, resulting from the juxtaposition of the two images, is 
provided in fig. 09: the false colours denote the difference between true and predicted classes. 
From the comparison between these data, the confusion matrix and related performance 
measures are also extracted. The rows and columns of the confusion matrix provide, for 
each class, the number of correctly and incorrectly classified pixels, respectively (fig. 09). 
The results show optimal values of accuracy and precision, on the order of 90%. 

In addition, compared to the manual approach, the supervised automatic classification 
allows to describe more details and differences in the material composition of the surfaces, 
and it requires a remarkably low amount of time (almost 30 minutes for the training and 
classification time against the manual annotation time of the whole orthophoto, which would 
require, to an estimate, from 6 to 9 hours). 


E 0- Brick masonry with traces 
of wall painting 
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Fig. 08. Validation set (a). 
Comparison between 
predicted (b) and true 
(manually-annotated, c) 
labels, and false colors 
map highlighting the 
differences between one 
class and another (d). 


Fig. 09. Confusion matrix 
and related performance 
measures extracted ba- 

sed on the validation set. 
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Average 90,41% | 90,16% | 96,14% | 90,27% | 


The control, by the operator, of the annotated data, and the eventual refinement, if needed, 
of some not correctly classified parts, could be further considered to optimize both results 
and classification times, still in the perspective of a supervised approach. 


Comparison of conservation states over time 


In this phase, the extra images are sub-aligned to the set of already aligned images, and the 
orientation parameters and shooting geometry of the cameras are consequently derived by 
appropriate insertion of control points, identified both in the added image as well as in the 
original ones (fig. 10). If the spatialization of the archival image is feasible, then the semantic 
information can be propagated from the newly added image to the rest of the (existing) 
image set, as well as to the 3D model. For the example of the Chapel of St. Agata, the 
insertion of additional images [Burresi, Caleca 2003], dating back to the 1960s, allows to 
characterize the original frescoes found on the inside cusp, as well as to compare them with 
the current situation of the walls, affected by advanced degradation patterns [Bevilacqua et 
al. 2017]. By evaluating the actual and previous states of the wall and by generating an orthophoto 
of the 3D model in the two respective conditions, a classification approach is applied to compare 
the mural traces of the 1960s and of today, respectively. The results are provided in fig. | |, and the 
segmentation of the archival image reveals the lost wall traces over the ortho-photo (fig. | | ©). 


Conclusions 


This work presented a classification-based methodology for the comparison between actual 
and previous states of a building. The comparison is derived by making use of Al- based 
semantic segmentation methods and by leveraging incremental photogrammetry for the 
spatialization of archival images on an already existing set of images. 


527 


Fig. 10. Spatialization 

of archival images. The 
input archival image (a) 

is suitably inserted within 
the existing -actual- pho- 
togrammetric model (b). 
The archival image is then 
trained (c) and classified 


Fig. II. Portion of the 
ortho-photo of the 
interior walls of the 
chapel: actual condition 
(a); spatialization of 

the archival image (b); 
evidence of previous wall 
traces highlighted by seg- 
mentation and projection 
of the archival image over 
the ortho-mosaic (c). 


E 0 - Brick masonry with traces 
of wall painting 


(| 1- Chromatic alteration of brick 
masonry 


The application to the case study of the Chapel of Sant'Agata in Pisa is representative of the 
methodology, as the integration between archival imagery, 3D surveying, and Al classification 
approaches allows to better describe and characterize the state of conservations of the 
building's interiors over time. The reality-based model of the Chapel is considered to compare 
the actual condition of the interior frescoes with the situation that existed at the time the 
frescoes were discovered, in the late 1960s. The temporal evolution of the frescoes is depicted 
by providing the evidence of previous wall traces, and by leveraging a semantic segmentation 
process. Considering the importance of the integration of such techniques, future work on this 
topic goes in the direction of a deeper fusion between 3D models, images, temporal states 
and classification techniques. In detail, the author is currently working on the construction 
of a dedicated application in which the reality-based model derived from surveying and the 
classification information derived from the study of archival documentation can be directly 
accessed, integrated and visualized. In such immersive experience, the integration and 
dissemination of reality-based models and classification results may be achieved by considering 
advanced real-time visualization tools as Unreal Engine (fig. | 2). 


E 0 - Brick masonry with traces 
of wall painting 


E1 1 - Chromatic alteration of brick 
masonry 
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Fig. 12. A digital rende- 
ring of the reality-based 
model of the Chapel via 
Unreal Engine, toward 
the integration and disse- 
mination of the classifica- 
tion results within Virtual 
Reality applications. 


Notes 


[I] This study is part of a larger research project, carried out in the framework of a Ph.D. thesis, on the semantic annotation 
transfer and retrieval for architectural heritage by combination of Artificial Intelligence, H-BIM and collaborative reality-based 
annotation platforms. The Ph.D. thesis involves, in the framework of a co-tutelle agreement, the Universities of Pisa and 
Florence, the Ecole Nationale Supérieure d'Arts et Métiers in Aix-en-Provence, the MAP laboratory (Modèles et Simulations 
pour l'Architecture et le Patrimoine) of the CNRS in Marseille and the Université Franco-Italienne. 

[2] The joint research project involving the Municipality of Pisa and the University of Pisa was documented in the M.Sc. thesis 
by Lavinia Tenchini [2017]. 
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